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CONFIDENTS 

1 

Method and Apparatus for Representing and Searching for an Object 

Using Shape 



The present invention relates to a method and apparatus for representing 
5 an object in a still or video image using shape, especially for use in searching. 
The invention also relates to a method and apparatus for searching for an object 
in a image using a shape representation. 

It is known to store still or video images, for example in image libraries, 
using representations of objects appearing in images. The representations are 
10 used in searching methods to enable images containing objects of interest to be 
. retrieved. The representations may be based on various features of the objects, 
including colour, texture and shape. 

Various methods for representing shapes of objects in images are 
known. Known methods include chain coding, the quad-tree method and the 
1 5 curvature scale space representation method. 

To perform a search in a image searching system, a user inputs a query 
by presenting the system with a sketch or image of the object sought, or by 
selecting a view of an object stored on the system. The system then derives or 
obtains a representation of the query object and compares the query 
20 representation with the representations of images stored in the database using 
an appropriate matching method. The closest matches are displayed on a 
display unit. 



2 

In most cases, an object as it appears In a video image is a projection of 
a 3-dimensionaI real object onto a 2 -dimensional image plane. The 2- 
dimensional shape or outline of an object appearing in an image thus depends 
on factors such as the viewing position, viewing angle and camera and optical 
5 system parameters. Objects will therefore have different outlines associated 
with different views. 

A disadvantage of known image database systems is that different 
views of an object having different outlines are treated as different objects. As 
a result, for example, if a user inputs a query based on a front view of an object 
10 A, and the object A only appears in a video sequence from the rear and side 
view, no successful matches will result and the object will not be retrieved. 

The present invention provides a method of representing an object 
appearing in a digital image comprising deriving representations of a plurality 
of different 2-dimensional views corresponding to the object, 
15 The invention also provides a method of matching objects comprising 

inputting a query and comparing the query with the plurality of representations 
of different views of objects to find the closest match or matches, by processing 
signals corresponding to images of objects. 

An embodiment of the invention will be described with reference to the 
20 accompanying drawings of which: 

Fig. 1 is a block diagram of a system according to an embodiment of 
the invention; 
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Fig. 2 is a diagram showing how a descriptor for an object is obtained; 
Fig. 3 is a flowchart illustrating how a descriptor for an objector is 
obtained; 

Fig. 4 is a diagram illustrating an apparatus and method for searching; 
Fig. 5 is a flowchart illustrating a search method. 

A system according to an embodiment of the invention is shown in Fig. 
L The system includes a control unit 2 such as a computer for controlling 
operation of the system, a display unit 4 such as a monitor, connected to the 
control unit 2, for displaying outputs including images and text, and a pointing 
device 6 such as a mouse for inputting instructions to the control unit 2. The 
system also includes an image database 8 storing digital versions of a plurality 
of video sequences and a descriptor database 10 storing descriptor information, 
described in more detail below, for objects appearing in each of the video 
sequences stored in the image database 8. Each of the image database 8 and 
the descriptor database 10 is connected to the control unit 2. 

In this embodiment, the elements of the system are provided on a single 
site, such as an image library, where the components of the system are 
permanently linked. 

A method of deriving a descriptor for an object according to an 
embodiment of the invention will be described with reference to Figs. 2 and 3. 
The method will be described for an object having a relatively simple shape, in 
this case a cylindrical object. 
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In this example, in a video sequence stored in the image database 8, the 
cylindrical object appears twice. Referring to Fig. 2, in the first appearance, 
the object outline corresponds to view 1, that is a view from the side and in the 
second appearance, the object outline corresponds to view 3, that is a 
5 perspective view from the side and above. 

The steps of the method will now be described with reference to the 
flowchart shown in Fig. 3. 

The database indexer recognises that the two outlines are of the same 
object and that the 3-dimensional object corresponding to those outlines is a 
10 cylinder (step 10). For each of view 1 and view 3, shape descriptors using the 
curvature scale space (CSS) representation are derived (step 20). Also, the 
. database indexer selects any additional views that are deemed representative of 
the object, even if they do not appear in the video sequence under consideration 
(step 30). In this example, view 2, which is a view of a cylinder from above, is 
15 selected as representative. A CSS representation of that view is also obtained 
(step 40), 

The paper "Robust and Efficient Shape Indexing through Curvature 
Scale Space" by Farzin Mokhtarian, Sadegh Abbassi and Josef Kittler, Proc. 
British Machine Vision Conference, pp. 53-62, Edinburgh, U.K. 1996 
20 describes a method of curvature scale space representation and is incorporated 
herein by reference. Briefly, a curve representing the outline of a shape is 
evolved by smoothing the curve. The curve is considered at a number of 
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different stages in the evolution. More specifically, the curvature zero 
crossings are identified in a curvature function at each of the stages of the 
evolution. A graph of the curvature zero crossings is obtained by combining 
zero crossing from all stages of the evolution. One axis of the graph 
5 corresponds to a which represents the evolution parameter of the curve and 
the other axis corresponds to a curve arc length parameter u. A shape is then 
represented by the locations of the maxima of the contours in the graph. 

In addition to CSS shape representations of representative views of the 
object, a global 3D shape parameter of the object, which is independent of the 

10 view of the object, is obtained (step 50). In this example, the global parameter 
is the volume of the object in real life. The volume may already be known, or 
it may be approximated with reference to other objects appearing in the video 
sequence, such as people, for which dimensions can be approximated. 

The global parameter and the view shape descriptors are combined to 

1 5 form a 3D object shape descriptor (step 60). 

3D object shape descriptors are obtained in the above manner for all 
objects of interest appearing in images in the image database 8. The number of 
views for any given object and which views are used depends on the 
complexity of the object. Some objects may have no global parameter in die 

20 3D object shape descriptor, for example, if a value for a volume is not known 
or not easily derived. Each view has a reference pointer indicating in which 
frame in the video sequence it appears, and also which object in the frame it is, 



for example, a view of an object may have a pointer indicating that it appears in 
frame 1000 and it is object number 3. 

In other embodiments using different shape representation methods, the 
number and nature of the views will also depend on the shape representation 
method used. For example, a method that is less sensitive to shape deformation 
resulting from changes in viewing geometry will require a smaller number of 

views. 

i 

The descriptor database 10 stores the 3D object shape descriptors for 
the objects in video sequences stored in the image database 8. 

A method of searching for an object in a video sequence will now be 
described with reference to Figs. 4 and 5. 

The user initiates a search by inputting a query. The query is input by 
using the pointing device 6 to draw one or more shape outlines on the display 
unit 4 (step 62). In this example, two query outlines 100 are input These are 
the query views. The user also inputs a parameter representing the volume of 
the object for which he is searching (step 64). 

The control unit 2 then derives the CSS view descriptor for each of the 
query views (step 66). 

In an alternative embodiment, the user inputs a queiy by selecting a 
query shape or shapes from a menu of shapes displayed on the display unit 4 by 
the control unit 2. In such an embodiment, the view descriptors may already be 
available in the descriptor database 10. 
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The query volume parameter and the query view descriptors are 
combined to form a query 3D object descriptor. 

The system then performs a matching procedure for detennining the 
similarity between the query object descriptor and the object descriptors stored 
5 in the descriptor database 10, hereinafter described as model object descriptors. 
Each model object descriptor in the descriptor database is selected in turn by a 

selector 205 (step 68) and the following steps are performed for each model 

i 

object descriptor in its turn. 

Firstly, a global similarity measure GS is obtained (step 70) by a 
10 comparator 200, using the global parameters for the query object descriptor and 
the model object descriptor. In this embodiment, the GS is derived by taking 
the ratio of the query volume parameter to the model volume parameter for the 
descriptor taken from the database. If the ratio is in the interval (1/c, c) where 
c>l, then the objects are considered similar and GS takes the value 0. 
15 Otherwise, GS takes the value infinity. The value of c depends on the 
application. For example, for feature film, c = 5. 

If at least one of the query object descriptor and the model object 
descriptor does not have a global parameter value, then GS = 0. 

If GS * 0 then a new model object descriptor is selected from the 
20 descriptor database; if GS = 0, then a view descriptor comparison is performed 
as follows (step 72). 
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Each query view descriptor is compared with each view descriptor for 
the model object descriptor under consideration in a comparator 810 using a 
matching function to derive view-similarity measures (step 74), A comparison 
of the ith query view descriptor and the jth model view descriptor results in a 
view similarity measure s^. 

In more detail, the query view descriptor is selected using a selector 600 
and the database view descriptor is selected using a selector 700. Firstly, the 
view descriptor for the first query view is compared with each view descriptor 
in the model object descriptor from the database. For each pair, a view- 
similarity value s is computed, using a suitable matching algorithm. In this 
embodiment, a view-similarity value s is computed using the matching 
algorithm described in the Mokhtarian, Abbasi and Kittler paper described 
above. When this particular similarity measure is used, the smaller the view 
similarity values, the closer is the match- This results in a set of k view- 
similarity measures for the first query view, where k is the number of view 
descriptors in the model object descriptor under consideration, and the k 
measures are stored in a global and local similarity combiner 820. 

Then view-similarity values are computed and stored for the second 
query view descriptor and the model view descriptors from the database in the 
same way, resulting in k further view-similarity measures. 

For each query view, the minimum view-similarity value for the 
database descriptor value under consideration is selected (step 76). This 



minimum value is a measure of the closest match between the respective query 
view and one of the views in the object descriptor being considered. This 
results in p minimum view-similarity values where p is the number of query 
views. In this example, p=2. 
5 An overall similarity measure S for the query descriptor and the model 

object descriptor being considered is taken as the median of the p similarity 
values (step 78). This represents the closeness of the match between the query 
descriptor and the model object descriptor taking all views into account. Thus, 
if one query view matches a view in the database descriptor closely but the 

10 other query view does not match any view in the database descriptor closely, 
then this is reflected in S by an intermediate value. 

The above steps are repeated for each object descriptor in the descriptor 
database 8, resulting in n similarity measures S, where n is the number of 3D 
object descriptors in the descriptor database (step 80). The n similarity 

15 measures are then ordered (step 82) starting from the lowest value which 
indicates the closest match- The m lowest values, where m is a value selected 
by the user or determined by the set-up of the control unit, are then selected, 
and an image from each of the corresponding m video sequences including the 
object is displayed on the display unit 4 (step 84). 

20 According to the invention, multiple views of a single object are stored 

to form a complete or quasi-complete description of the outline shape. The 
representation may be quasi-compLete if only views considered important for 
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retrieval are stored. For example, for a typical database storing feature films, 
only front, back, side and top views of a car may be stored as a unified 
representation, but the view from under the car is not stored, as it is unlikely 
that that view will be used as a query. 

A system according to the invention may, for example, be provided in 
an image library. Alternatively, the databases may be sited remote from the 
control unit of the system, connected to the control unit by a temporary link 
such as a telephone line or by a network such as the internet. The image and 
descriptor databases may be provided, for example, in permanent storage or on 
portable data storage media such as CD-ROMs or DVDs. 

Components of the system as described such as the selectors and 
comparators may be provided in software or hardware form. Although the 
invention has been described in the form of a computer system, it could be 
implemented in other forms, for example using a dedicated chip. 

Specific examples have been given of methods of representing a 2D 
shape of an object and of methods for calculating values representing 
similarities between two shapes but any suitable such methods can be used. 

The various views of an object making up the complete or quasi- 
complete shape description may, for example, be provided by the film 
producer. For example, in a film including a car, the producer may arrange for 
20 different views of the car to be shot for use in indexing the film in the 
database. Alternatively, a link may be provided between all the different views 
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of an object in a sequence to enable the views to be found, and the shape fox 
any useful views not appearing in the sequence may, for example, be provided 
when data for the first view of the object is included. 

The invention can also be used, for example, for matching images of 
objects for verification purposes, or for filtering. 



Claims 
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1 . A method of representing an object appearing in a still or video 
image, wherein the object appears in the image with a first two-dimensional 

5 outline, by processing signals corresponding to the image, the method 
comprising deriving a view descriptor of the first outline of the object and 
deriving at least one additional view descriptor of the outline of the object in a 
different view, and associating the two or more view descriptors to form an 
object descriptor. 

10 

2. A method as claimed in claim 1 wherein the view discriptors are 
derived using a curvature scale space representation. 

3. A method as claimed in claim 1 or claim 2 further comprising' 
15 deriving a descriptor of the object which is related to the shape and/or size of 

the object and which is independent of the view of the object in the image. 

4. A method of representing an object appearing in a still or video 
image comprising deriving a descriptor which is related to the shape and/or size 

20 of the object and which is independent of the view of the object in the image. 
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5. A method as claimed in claim 3 or claim 4 wherein said view- 
independent descriptor corresponds to the volume of the object 



6. A method of searching for an object in still or video images by 
5 processing signals corresponding to images, the method comprising inputting a 

query to the computer in the form of at least one two-dimensional outline of an 
object, deriving a descriptor of the query object, comparing said query 
descriptor with stored descriptors for objects in images derived in accordance 
with a method as claimed in any one of claims 1 to 4 and selecting and 
10 displaying at least one result corresponding to an image containing an object 
for which the comparison between the respective descriptor and the query 
descriptor indicates a degree of similarity between the query and said object. 

7. A method as claimed in claim 6 dependent on claim 1 or claim 2 
2 5 wherein a query is input in the form of two or more two-dimensional outlines 

of an object, and wherein a query view descriptor is derived for each said 
outline, and wherein the step of comparing comprises comparing each said 
query view descriptor with each view descriptor in each stored object 
descriptor to derive a plurality of view-similarity values. 

20 

8. A method as claimed in claim 6 wherein the view-similarity 
values are analyzed to derive object similarity values. 
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9. A method as claimed in any one of claims 6 to 8 wherein at least 
some of the object descriptors include view-independent descriptors derived in 
accordance with a method as claimed in claim 3 and wherein the method 
comprises inputting a view-independent query value and the 'step of comparing 

5 compares the query value with the view-independent descriptors for the stored 
object descriptors, 

10. A method as claimed in any one of claims 6 to 9 wherein the 
query descriptor is derived using a curvature scale space representation of the 

1 0 query object outline. 

11. A method of representing an object appearing in a image by 
processing signals corresponding to said image, the method comprising 
deriving representations of a plurality of different 2-dimensional views' 

1 5 corresponding to the object. 

1 2. An apparatus adapted for implementing a method as claimed in 
any one of claims 1 to 1 1 . 
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13. A computer program for implementing a method as claimed in 
any one of claims 1 to 1 1 . 
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14. A computer system programmed to operate according to a 
method as claimed in any one of claims 1 to 1 1 . 

15. A computer-readable storage medium storing computer- 
executable process steps for implementing a method as claimed in any one of 
claims 1 to 1 1 . 

16. A method for searching for objects in still or video images 
substantially as hereinbefore described with reference to the accompanying 
drawings. 

17. A computer system substantially as hereinbefore described with 
reference to the accompanying drawings. 
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Method and Apparatus for Representing and Searching for an Object 

Using Shape 

5 Abstract 

\ A method of representing an object appearing in a still or video image 
for use in searching, wherein the object appears in the image with a first two- 
dimensional outline, by processing signals corresponding to the image, 
10 comprises deriving a view descriptor of the first outline of the object and 
deriving at least one additional view descriptor of the outline of the object in a 
different view, and associating the two or more view descriptors to fonn zn 
object descriptor. 
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of Title 35, United States Code Section 112, 1 acknowledge the duty 
to disclose information which is material to patentability as defined in 
Title 37, Code of Federal Regulations, Section 1 .56 which became 
available between the filing date of the prior application and the 
national or PCT International filing date of application. 

(Status: Patented, Pending, Abandoned) 
(Status: Patented, Pending, Abandoned) 

mu : »wfrni«N mmm) 

I hereby declare that all statements made herein of my own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the 
like so made are punishable by fine or imprisonment, or both, under 
Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the validity of the application 
or any patent issued thereon. 
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Japanese Language Declaration 



POWER OF ATTORNEY: As a named inventor, I hereby appoint 
the following attorney(s) and/or agent(s) to prosecute this 
application and transact all business in the Patent and Trademark 
Office connected therewith (list name and registration number) 



TERRELL C. BIRCH (Reg. No. 19,382] 
RAYMOND C. STEWART {Reg. No. 2T.066} 
JOSEPH A. KOLASCH (Reg. No. 22.4631 
ANTHONY L. BIRCH (Reg. No. 26,122) 



JAMES M. S LATTERY (Reg. No. 28.3801 
BERNARD L SWEENEY (Reg. No. 24,448) 
MICHAEL K. MUTTER (Reg. No. 29,680) 
CHARLES GORENSTEiN (Reg. No. 29,271) 



GERALD M. MURPHY (Reg. No. 28.977) 
LEONARD R. SVENSSON {Reg. No. 30,330) 
TERRY L CLARK (Reg. No. 32,644} 
ANDREW 0. MEIKLE {Reg. No. 32,868) 



MARC S. WEiNER (Reg. No, 32,181) 
ANDREW F. REISH (Reg. No. 33.443) 
JOE M. MUNCY (Reg. No. 32,334! 
C. JOSEPH FARACI (Reg. No. 32,350) 



Send Correspondence to: 



BIRCH. STEWART, KOLASCH & BIRCH, LLP 
P.O. BOX 747 

FALLS CHURCH, VA 22040-0747 
TEL: (703) 205-8000 



Direct Telephone Calls to: (name and telephone number) 



BIRCH, STEWART, KOLASCH & BIRCH, LLP 
TEL: {703! 205-8000 



Full name of sole or first inventor 
Miroslaw Z. Bober 



Residence 



Surrey, U.K 




Date 



4 AKl3o ap 



Citizenship 



Poland 



Post Office Address 

18-20, Frederick Sanger Road. The Su 



Research Park, Guildford, Surrey 
GU2 6 YD U.K. 



Full name of second joint inventor, if any 



55" 



Second inventor's signature 



Residence 



Date 



Citizenship 



Post Office Address 



(Supply similar information and signature for third and subsequent 
joint inventors.) 
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